Cognitive Architecture as Revolutionary Science

Many proposals for evaluating cognitive architectures rely partly on an analogy between cognitive science and other sciences. We therefore reflect on the relations between cognitive science, cognitive architectures and other sciences. Some important differences caution against the misguided use of some evaluation approaches. Some unique characteristics of human intelligence motivate the need for new approaches to evaluating cognitive architecture design. We briefly propose one such approach. In science there is a general expectation that when you do something, you have some way of showing that it has achieved your goal, that it is in some way valid. For example, when a theory generates predictions that other theories do not, then confirming these predictions is evidence it is true. Another way to support a theory is true is to prove theorems about some of its desired characteristics. These approaches successfully used by so many scientific communities that people naturally expect cognitive modelers to adopt them. This is based on a misunderstanding of both science and of the cognitive architecture approach. First, let us recall the distinction (Kuhn, 1970) between revolutionary science and normal science. These can be illustrated with an example. Many scientists working within the Newtonian framework accept Newton’s conception of space, time, inertia, mass and force and also accepted his three laws regarding the relations between these. Their work involves applying this apparatus to generate explanations or predictions of specific phenomena. They conduct experiments to confirm that the observations their work leads them to expect are indeed observed. If an experiment fails, they generally assume it is a problem with their application of the Newtonian framework instead of a flaw in the framework itself. Khun calls this normal science. He calls the project of developing a framework such as Newtonian physics, molecular biology or evolutionary biology revolutionary science. As Kuhn, Quine (1951) and others point out, one does not confirm or disconfirm a scientific framework with one or a small number of experiments. When a particular prediction of a framework fails, one normally first blames their application of the framework. For example, clouds and birds do not obviously obey Newton’s laws. Rather than rejecting Newtonian physics, one thinks very carefully about the specific forces involving cloud and bird motion and attempts to conceive of them so that they are consistent with Newton’s laws. When scientists accepting Newtoinan physics observed Uranus and Neptune’s orbits were not what they expected, they did not reject Newtonian physics but instead posited the existence of a large mass disturbing the orbit. This mass turned out to be Pluto. These examples illustrate that the observation-hypothesis-experiment caricature of the scientific method does not apply to developing a scientific framework. It is our thesis that developing cognitive architectures has some of the characteristics of revolutionary science and that developing models within a cognitive architecture has more of the characteristics of normal science. A cognitive architecture provides a basic computational framework for explaining and predicting human cognition. For example, in many architectures, rules, declarative memory and a process for using these to select actions function like the concepts of mass, inertia and Newton’s laws do in physics. They provide a basic framework for explaining and predicting a wide range of phenomena. Models of cognition in a particular situation are similar to explanations of specific physical phenomena in that they use the basic apparatus of the conceptual framework without disrupting it. The fact that cognitive architectures are more like a scientific framework than a specific explanation of a phenomena has several implications. First, just as specific experiments are not sufficient to evaluate scientific frameworks, cognitive architectures cannot be evaluated or compared merely by employing the traditional model fitting or hypothesis testing from experimental psychology. For example, when an ACT-R model makes an incorrect prediction it is more likely a problem with a production rule, parameter or declarative memory element in the model than the architectural framework of ACT-R itself. How then should cognitive architectures be evaluated? Typical standards used to evaluate other scientific frameworks are: Do they explain and predict a wide range of phenomena that were previously unexplained? Are they